Apparatus and method for predicting/recognizing occurrence of personal concerned context

ABSTRACT

Disclosed is technology for providing a proper UI/UX through various devices or services when occurrence of registered concerned context is predicted or recognized in order to predict or recognize the circumstances that require attention or emotion control with regard to a change in his/her biological information With this, a user designates his/her own biological information range or emotional state with regard to circumstances which catch his/her attention, and registers concerned context by selectively designating attributes of circumstantial elements. Further, a user registers feedback desired to be given and an external device/service desired to interface with when the occurrence of the concerned context is predicted or recognized. According to the attributes of the circumstances designated in the registered concerned context, points in time for collecting and managing UX data are automatically determined, thereby processing and managing the UX data as useful information.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of Korean PatentApplication No. 10-2018-0062753, filed on May 31, 2018, the disclosureof which is incorporated herein by reference in its entirety.

BACKGROUND 1. Field of the Invention

The present invention relates to technology for giving a user a feedbackon concerned context through various devices or services when occurrenceof registered concerned context is predicted or recognized under acondition that the user has designated his/her own biologicalinformation, emotional state, and attributes of circumstances (time,place, behavior, etc.) and has registered the concerned context thatrequires attention or emotion control with regard to the designatedattributes.

2. Discussion of Related Art

Researches and development have been steadily carried out onlife-logging for providing a useful service to a user by storing andanalyzing personal daily experience data. The life-logging based servicehas recently been required to provide a function of actively improvingthe quality of physical and emotional life to a user based on his/herexperience data as well as a simple function of monitoring a user'sbehavior or setting an alarm. That is, when a user needs to controlemotions or be careful about a change in his/her biological information(e.g., a biological signal), there is a need of providing a proper UI(user interface)/UX (user experience) suitable for a time and space inwhich the user is present. Further, there is a need of determining aproper point in time to process the experience data, and extracting andmanaging information useful for grasping personal characteristics fromthe continuously generated experience data.

Human emotions are closely related to their voice and/or biologicalinformation. For example, in the case of a negative emotional state suchas an angry state, voice becomes louder, a tone becomes higher, a heartrate increases, breathing becomes faster, body temperature rises, and/ortension in muscles increases as compared to those in a usual state.Further, in the case of a negative emotional state such as a depressedstate, a user voice generally features slow speaking, a long pause,small pitch variation, and a low amplitude.

In addition, the voice and/or the biological information are closelycorrelated with an ambient temperature and a user's specific behaviorssuch as sitting down, walking, standing up, running, sleeping, his/herspecific utterance, etc. For example, a user's heart rate and breathingrate in a dynamic state such as walking or running become faster thanthose in a static state such as sitting down or standing up, and voicefeatures are also changed in between. For example, a user's pulse ratemay measure higher than usual when she/he gets emotionally excited, butstrenuous exercise or high atmospheric temperature may also make thepulse rate become higher. As an example of a correlation between theatmospheric temperature and the biological information, blood vesselsbeneath a human skin is dilated at a high temperature and skintemperature rises, thereby causing a biological response of giving offbody heat. In other words, a heart increases the amount of bloodcirculated beneath the skin and causes a rapid pulse and an increasedcardiac output, thereby making a heartbeat rapid.

Therefore, not only the correlation between the emotion and the voiceand/or the biological information but also the correlation among theuser behavior, the atmospheric temperature, and the voice and/or thebiological information have to be taken into account in order toaccurately recognize the emotional state among pieces of informationabout the concerned context registered by a user. Further, such featureshave to be extracted and managed based on personal experienceinformation since they largely vary depending on individuals.

SUMMARY OF THE INVENTION

An aspect of the present invention is to provide a proper UI/UX througha feedback as desired by a user by learning-based prediction orrecognition of circumstances, which requires physical or emotionalattention, registered by the user based on his/her experience data fromreal-life environments.

The present invention is conceived to solve the above problems, anddirected to proposing technology for providing a UI/UX through variousdevices or services when occurrence of registered concerned context ispredicted or recognized under a condition that a user has designatedhis/her own biological information range, emotional state, andcircumstances such as time, place, behavior, etc. to be registered asthe concerned context in order to predict or recognize the circumstancesthat require attention or emotion control with regard to a change inhis/her biological signal (biological information).

With the apparatus and method for predicting/recognizing occurrence ofpersonal concerned context according to one embodiment of the presentinvention, a user designates his/her own biological information range oremotional state with regard to circumstances which catch his/herattention, and registers concerned context by selectively designatingattributes of circumstantial elements such as time, place, behavior,atmospheric temperature, etc. Further, a user registers a feedbackdesired to be given and an external device/service desired to interfacewith when the occurrence of the concerned context is predicted orrecognized. Depending on the match between the current circumstance andthe attributes of the circumstances designated in the registeredconcerned context, points in time for collecting and managing UX dataare automatically determined, thereby processing and managing the UXdata as useful information. Further, a personal emotion-biologicalinformation model of the present invention is configured to be robust toinfluences of a user behavior and an atmospheric temperature whilerecognizing his/her biological information range and emotional state,thereby having functions for generating, configuring and managing thepersonal emotion-biological information model capable of predicting andrecognizing the occurrence of the registered concerned context.

Further, a mode of predicting/recognizing occurrence of personalconcerned context is provided to calculate an occurrence probability ofthe concerned context by automatically recognizing a user's biologicalinformation range and emotion on the basis of experience data featureinformation of the collected voice and biological information (i.e., abiological signal) and generate and interface a concerned contextoccurrence prediction or recognition event according to the calculatedoccurrence probability.

According to one aspect of the present invention, there is provided anapparatus for predicting/recognizing occurrence of personal concernedcontext, the apparatus including: a UI device including a concernedcontext definer through which a user designates biological information,emotions, and circumstances and registers concerned context about theuser's own biological information change or emotional state; and anemotion-biological information model management module including anemotion-biological information model manager which reflects informationabout the concerned context registered by the user in anemotion-biological information model, makes the emotion-biologicalinformation model learn from a user voice/biological information featureand reference statistics of the user voice/biological informationfeature depending on user behavior and atmospheric temperature, andmanages the emotion-biological information model, and aconcerned-context event generator which generates a concerned-contextoccurrence prediction event or a concerned-context occurrencerecognition event by predicting or recognizing the occurrence of theconcerned context registered in the concerned context definer of the UIdevice. According to one embodiment, the reference statistics about avoice feature depending to the user behavior and the atmospherictemperature may include maximum, minimum, average, deviation, or similarinformation about the voice/biological information feature extractedeither in a frequency domain such as a pitch or in a time domain such asspeaking rate (tempo) or change in amplitude of energy.

A user may designate a change in “biological information,” states of“emotions,” and attributes of “circumstances” such as time, place, userbehavior, and atmospheric temperature, etc. to thereby register, modify,and manage a specific concerned context c_(k) which requires his/herattention or is desired to be controlled. For example, a user maydesignate the concerned context to be registered by him/her, with aconcerned context label (in the form of a descriptive text set by auser) such as “increasing heart rates in a crowded place”, “angry atconversation”, etc.

For example, as shown in the following Table 1 (with sorts of concernedcontext, elements, and user-designated attributes), an absolute timerange based on the universal standard time or semantic attributes suchas ‘morning’, ‘afternoon’, or ‘evening’ may be designated as “attributesof time”, i.e., elements that belong to “circumstances” in the concernedcontext c_(k). Further, an absolute address or semantic attributes suchas ‘home’, ‘street’, and ‘public place’ may be designated as “attributesof place” in the circumstances. In addition, general behaviors such as‘sitting down’, ‘lying down’, ‘walking’, ‘running’, etc. or specificbehaviors such as ‘sleeping’, ‘conversation’, ‘utterance of specificwords’, etc. may be designated as “attributes of user behavior”.

TABLE 1 Examples of attributes designated Sort Elements by userCircumstances Time Morning, afternoon, evening Place Address, home,street, public place Atmospheric — temperature Behavior Sitting down,standing up, walking, running, conversation, sleeping, specificutterance Biological Heart rates Lower, normal, higher, regular,information HRV irregular GSR Respiration volume Body temperatureEmotions Voice/biological Neutral, positive, negative informationfeatures

The elements included in the “biological information” include a heartrate (HR), a heart rate variation (HRV) calculated based on heartbeatinformation, a galvanic skin response (GSR), a respiration volume, abody temperature, etc., and the attributes such as ‘normal’, ‘lower’,‘higher’, ‘regular’, ‘irregular’, etc. may be designated to theseelements.

Further, the elements included in the “emotion” include a voice featureand a biological information feature, and the attributes such as‘neutral’, ‘positive’ and ‘negative’ may be designated to theseelements. In addition, detailed attributes corresponding to emotionssuch as sadness, anger and the like may be designated to ‘negative’.

Meanwhile, for the registration of the concerned context and theconfiguration of the personal emotion-biological information model, aset C of concerned contexts c_(k) registered by a user, a set S ofelements s_(n) in each circumstance, a set B of biological information(or biological signals) b_(m), and a set E of recognized emotion labelse_(j) may be defined by the following Table 2.

TABLE 2 Set of concerned contexts c_(k) C = {c₁, c₂, . . . , c_(k)}, k ≤K Set of elements s_(n) of circumstance S = {s₁, s₂, . . . , s_(n)}, n ≤N Set of biological signals b_(m) included in B = {b₁, b₂, . . . ,b_(m)} m ≤ M biological information Set of recognized emotion labelse_(j) E = {e₁, e₂, . . . , e_(j)}, j ≤ J

When a user registers the concerned context, there is no limit to waysof designating the detailed attributes of each element. For example, auser may not designate the attributes of a place or specific biologicalinformation range, but designate an attribute of ‘morning’ as theelement of time in the concerned context, an attribute of ‘conversation’as the element of user behavior, and an attribute of ‘negative’ as theelement of emotion, thereby registering the concerned context with aconcerned context label of “negative emotion at conference.”

According to one embodiment of the present invention, the UI device mayfurther include a circumstance extracting and recognizing unit whichextracts a current circumstance received from the UI device or a userwearable information collection device, and recognizes whether concernedcontext designated with a circumstance matching the extracted currentcircumstance has been registered; and a voice/biological informationfeature extractor which extracts a voice/biological information featurefrom current user voice/biological information and transmits theextracted voice/biological information feature along with thecircumstance to the emotion-biological information model managementmodule when the circumstance extracting and recognizing unit recognizesthat the current circumstance matches the circumstance of the registeredconcerned context.

According to one embodiment of the present invention, the UI device mayfurther include a feedback registration unit which registers a type offeedback desired to be given when the concerned-context occurrenceprediction or recognition event is generated; and a feedback giver whichgives the registered type of feedback when the concerned-contextoccurrence prediction or recognition event is received from theemotion-biological information model management module.

According to one embodiment of the present invention, the UI device mayfurther include an external device/service registration unit whichregisters an external device and service to interface with the feedbackgiven when the concerned-context occurrence prediction or recognitionevent is generated.

According to one embodiment of the present invention, theemotion-biological information model management module may furtherinclude an emotion recognizer which extracts a feature vector to be usedfor recognizing the user's emotion from the voice/biological informationfeature received from the UI device; and a concerned-context occurrenceprobability calculator which calculates an occurrence probability of theconcerned context on the basis of the emotion-biological informationmodel and the extracted voice/biological information feature vector.

According to one embodiment of the present invention, theconcerned-context event generator of the emotion-biological informationmodel management module may use a threshold for determining occurrenceprediction and a threshold for determining occurrence recognition so asto determine either of concerned-context occurrence prediction orconcerned-context occurrence recognition.

According to another aspect of the present invention, there is providedan apparatus for predicting/recognizing occurrence of personal concernedcontext, the apparatus including: a concerned context definer throughwhich a user designates biological information, emotions, andcircumstances and registers concerned context about the user's ownbiological information change or emotional state; an emotion-biologicalinformation model manager which reflects information about the concernedcontext registered by the user in an emotion-biological informationmodel, makes the emotion-biological information model learn from a uservoice/biological information feature and reference statistics of theuser voice/biological information depending on user behavior andatmospheric temperature, and manages the emotion-biological informationmodel; and a concerned-context event generator which generates aconcerned-context occurrence prediction event or a concerned-contextoccurrence recognition event by predicting or recognizing the occurrenceof the concerned context registered in the concerned context definer.

According to still another aspect of the present invention, there isprovided a method of predicting/recognizing occurrence of personalconcerned context, the method including: by a user, designatingbiological information, emotions, and circumstances and registeringconcerned context about the user's own biological information change oremotional state; reflecting information about the concerned contextregistered by the user in an emotion-biological information model,making the emotion-biological information model learn from at least oneof user voice and biological information features and referencestatistics of user biological information feature depending on userbehavior and atmospheric temperature, and managing theemotion-biological information model; and generating a concerned-contextoccurrence prediction event or a concerned-context occurrencerecognition event by predicting or recognizing the occurrence of theregistered concerned context.

The configuration and effect of the present inventive concept introducedas above will become apparent by the detailed description set forthherein with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentinvention will become more apparent to those of ordinary skill in theart by describing exemplary embodiments thereof in detail with referenceto the accompanying drawings, in which:

FIG. 1 is a conceptual view of a system for predicting/recognizingoccurrence of personal concerned context;

FIG. 2 is a conceptual view of service in a system forpredicting/recognizing occurrence of personal concerned context;

FIG. 3 is a view illustrating functional elements in a system forpredicting/recognizing occurrence of personal concerned context;

FIG. 4 shows an example of how voice/biological information featurevectors are input to a model for learning the voice/biologicalinformation feature vectors;

FIG. 5 shows a user interface (UI) screen for registering and modifyingconcerned context;

FIG. 6A and FIG. 6B are flowcharts showing operations of a system forpredicting/recognizing occurrence of concerned context; and

FIG. 7 is a configuration view of an emotion-biological informationmodel.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 1 is a schematic configuration view for explaining a detailedembodiment of an apparatus and method for predicting/recognizingoccurrence of personal concerned context according to the presentinvention. This embodiment includes information collection devices 110-1and 110-2 attached to, worn on, or carried by a user 100, a userinterface (UI) device 120 such as a smart phone, etc., and anemotion-biological information model management module 130 configurablein an additional independent server (including a cloud server) andinstallable even in the UI device 120.

Through the UI device 120, the user 100 may register concerned contextin the form of a label (or a descriptive text) by designating his/herown concerned context label, specific biological information range,emotional state, and attributes of circumstances such as time, place,behavior, etc., and modify and manage the concerned context.

In the concerned context registered by the user 100, informationcollectable through the information collection devices 110-1 and 110-2includes biological response information such as a heartbeat, heart ratevariation (HRV), galvanic skin response (GSR), respiration volume, bodytemperature, etc. and circumstances such as a user behavior, etc. (auser's behavior information is collectable using an acceleration sensoror the like as the information collection device). Further, if awireless microphone is used as the information collection devices 110-1and 110-2, it is possible to collect a user's voice (actually, the UIdevice 120 is capable of autonomously collecting the voice). One or aplurality of information collection devices 110-1 and 110-2 may beemployed, and therefore the voice, the biological information, thecircumstance, etc. of the user 100, which are collected in each of thedevices 110-1 and 110-2, may also be collected as one or a plurality ofsignals (e.g., a voice signal, a biological signal, a motion signal,etc.). In the present invention, there are no limits to the position atwhich the information collection device acquires information, the numberof devices, and the kind of information to be collected.

The UI device 120 determines whether the circumstances such as time,place, user's behavior, atmospheric temperature, etc., which areextracted from data received from the information collection devices110-1 and 110-2 or which the UI device 120 autonomously detects, matchthe attributes of circumstantial elements in the concerned contextregistered by the user 100.

Further, the UI device 120 extracts at least one of a voice feature anda biological information feature (hereinafter, referred to as a‘voice/biological information feature’) from at least one of the voiceand the biological response information of the user (hereinafter,referred to as ‘voice/biological response information’) which aredetected from the data received from the information collection devices110-1 and 110-2 or which the UI device 120 autonomously detects; andtransmits the voice/biological information feature, along with theextracted or detected circumstance, to the emotion-biologicalinformation model management module 130. In this case, a user'svoice/biological information feature extracted by the UI device 120 issampled in units of a predetermined window Win_(thr) (see FIG. 4 and itsexplanation). For example, the voice feature includes a feature in afrequency domain, such as mel-frequency cepstrum coefficients (MFCC),and a feature in a time domain, such as an amplitude, a speaking rate,an emphasis, pitch variability, utterance duration, a pause time, etc.of a voice. Likewise, each biological information feature is alsosampled in units of the window Win_(thr) (see FIG. 4 and itsexplanation). For example, the biological information feature mayinclude the maximum, minimum, average, deviation and similar features ina time section.

The emotion-biological information model management module 130automatically recognizes a user's biological information range andemotional state from the circumstance and the voice/biologicalinformation feature received from the UI device 120 on the basis of analready learned emotion-biological information model (e.g., a user'semotional state is inferable from the emotion-biological informationmodel has been learned with the voice, biological information, andcircumstance of the user), and calculates a probability of occurrence ofconcerned context so as to recognize the occurrence of the registeredconcerned context (when the concerned context has already occurred) orpredict the occurrence of the registered concerned context (when theconcerned context is going to occur or gradually completing theoccurrence). Thus, when the occurrence of the registered concernedcontext is predicted or recognized, an occurrence prediction event or anoccurrence recognition event (hereinafter, also inclusively referred toas ‘the concerned context occurrence event’) of the correspondingconcerned context is generated and transmitted to the UI device 120and/or an external device/service 140 such as a personal computer (PC),an artificial intelligence (AI) device, an Internet-of-things (IoT)device, other dedicated devices, a social media service, or the likeregistered by a user with regard to the corresponding concerned context,or a registered service interface is called. The concerned contextoccurrence event involves the identifier of the corresponding concernedcontext, the occurrence prediction or recognition time, and the typeidentifiers of the occurrence prediction event/occurrence recognitionevent.

FIG. 2 illustrates a service concept of an apparatus and method forpredicting/recognizing occurrence of personal concerned contextaccording to one embodiment of the present invention. Basically, a userregisters concerned context and uses a learned emotion-biologicalinformation model to receive feedback on the occurrence of the concernedcontext through his/her own interface device (e.g., a smartphone, etc.)or an external device/service registered by him/her when the occurrenceprediction or recognition event of the concerned context occurs.

Referring to FIG. 2, a service scenario according to one embodiment ofthe present invention will be described in detail. First, it will beassumed that a user registers a circumstance, in which a range ofheartbeat biological information is higher than usual and an emotionalstate is ‘negative’ in a ‘room (place)’ at ‘night (time)’, with aconcerned context label of ‘a cautionary circumstance about a heartbeatand an emotion’, and designates/registers the concerned context to beposted on an information posting board of a certain social media service(corresponding to an external device/service 140) along with vibrationand text display in the UI device 120 as feedback to be received whenthe occurrence prediction or recognition event is generated.

When a specific element (e.g., a concerned-context event generator 234)of a module for managing the emotion-biological information model (See130 in FIG. 1) generates a concerned context occurrence prediction orrecognition event, this event is transmitted to the UI device 120.Therefore, a user receives feedback, which is given in the form of thevibration and the text registered by him/her, through the UI device 120(or through the information collection device 110) and thus can predictor recognize the occurrence of the corresponding concerned context.Further, the user can call a registered external service interface (forexample, a social media service information posting board) and check theoccurrence of the concerned context through his/her external device 140.

In such a manner, a user receives a UI/UX for feedback suitable to atime and space that s/he is in, through the UI device 120 and/or theexternal device/service 140 in accordance with types (occurrenceprediction or recognition) of a concerned context event that occurs inthe emotion-biological information model management module 130.

FIG. 3 is a configuration view of an apparatus and method forpredicting/recognizing occurrence of personal concerned contextaccording to one embodiment of the present invention. Elementsillustrated in FIG. 3 and the following drawings and descriptions referto process performing steps in light of a method invention, but may alsorefer to process performing materializers in light of an apparatusinvention.

The information collection device 110 includes a biological informationcollector 211 for collecting biological information by detecting abiological signal such as a heartbeat, an HRV, a GSR, a respirationvolume, a body temperature, etc. from a user's body. According toanother embodiment, the information collection device 110 mayadditionally include a circumstance collector 212 for collectingcircumstances about behaviors such as a user's walking, running, lyingdown, sleeping, specific utterance, etc.

Through a concerned context definer 221 of the UI device 120 (e.g., apersonal portable terminal such as a smartphone or the like), a user mayperform functions of registering, modifying and deleting the concernedcontext. Further, through an external device/service registration unit222, a user can register and manage the external device/service 140 forinterfacing feedback to be received when a prediction or recognitionevent corresponding to the occurrence of the concerned context isgenerated. For the registration, an address of an external device or anapplication programming interface (API) address of an external servicesuch as a social media service or the like may be designated. Throughthe external device/service registration unit 222, a user may receivefeedback on a proper UI/UX service through his/her own UI device 120and/or from various external devices and/or services 140 when theoccurrence of the concerned context is predicted or recognized.

Further, the UI device 120 includes a feedback registration unit 223 forregistering a feedback type such as a text, a sound, vibration, light,etc. desired to be given when a prediction or recognition eventcorresponding to the occurrence of the concerned context is generated,and a feedback giver 224 for giving the registered feedback type (e.g.,displaying a text, making a sound/vibration, turning on a light-emittingdiode (LED), etc.) when the prediction or recognition eventcorresponding to the occurrence of the concerned context is receivedfrom the emotion-biological information model management module 130.

In addition, a circumstance extracting and recognizing unit 225 of theUI device 120 extracts current circumstances such as a time slot, aplace, a user behavior, an atmospheric temperature, etc. from the UIdevice 120 itself, and recognizes whether concerned context designatedwith a circumstance matching the currently extracted circumstance hasbeen registered. To extract semantic attribute values of a place, atime, and a behavior, the UI device 120 may use a built-in timer and abuilt-in sensor, or information provided by other services (weatherapplication, an address conversion application, etc.) driven in the UIdevice 120. In an embodiment where the user-wearable informationcollection device 110 includes a separate circumstance collector 212, aspecific circumstance may be received from the circumstance collector212.

To determine the semantic attribute values (for example, to make piecesof set information match semantic attributes of absolute time when auser sets ‘time’ with an evening time slot, a dawn time slot, etc.), atechnique based on rules or probabilistic inference may be employed.

A voice/biological information feature extractor 226 of the UI device120 extracts voice/biological information features from a current uservoice (acquirable by automatically activating a built-in microphone ofthe UI device 120) and current user biological information (receivedfrom the biological information collector 211) when the circumstanceextracting and recognizing unit 225 recognizes that the currentcircumstance matches the circumstance of the registered concernedcontext, and transmits the extracted voice/biological informationfeatures along with the circumstance to the emotion-biologicalinformation model management module 130 through an emotion-biologicalinformation model interface 227.

Meanwhile, the emotion-biological information model management module130 includes an emotion-biological information model manager 231 thatreflects concerned context information registered by a user in anemotion-biological information model, and controls theemotion-biological information model to learn both voice/biologicalinformation feature generated for emotions and reference statistics of auser's biological information feature according to atmospherictemperatures and user behavior. Further, an emotion recognizer 232 isprovided to generate a feature vector to be used in recognizing a user'scurrent emotional state by reflecting reference statistics of biologicalinformation according to the user's behavior/atmospheric temperature,from voice/biological response information received from the UI device120. Further, a concerned-context occurrence probability calculator 233calculates a concerned-context occurrence probability corresponding to arecognized biological information range and a recognized emotionattribute.

The emotion recognizer 232 and the concerned-context occurrenceprobability calculator 233 determine a user's biological informationrange by using the already learned emotion-biological information modeland the generated voice/biological information feature information,thereby recognizing the emotional state and calculating theconcerned-context occurrence probability. In this case, theconcerned-context occurrence probability may reflect a probability to berecognized as a specific emotion label during an emotion recognitionprocess.

The concerned-context event generator 234 compares the calculatedconcerned-context occurrence probability with a predetermined thresholdto determine ‘concerned-context occurrence prediction’ or‘concerned-context occurrence recognition’ and generate eachcorresponding event. There may be various determination methods, and oneof them is to compare a first threshold thr₁ for determining the‘occurrence prediction’ and a second threshold thr₂ for determining the‘occurrence recognition’ to thereby predict or recognize the occurrenceof the concerned context. For example, it may be designed to predict theoccurrence when the concerned-context occurrence probability is higherthan the first threshold but lower than the second threshold, and torecognize the occurrence when the probability is higher than the secondthreshold. Alternatively, it may be designed to determine the occurrencerecognition rather than the occurrence prediction when an increasingtrend of the probability having a predetermined pattern is detectedbetween the first threshold and the second threshold by considering aprobability calculation cycle (e.g., 1 second, 5 seconds, 10 seconds, .. . ). Besides, a predetermined criterion may be made by various methodsso as to determine the concerned-context occurrence prediction orrecognition.

Meanwhile, as described above, the emotion-biological information modelmanagement module 130 (including the emotion-biological informationmodel) may be configured in a separate independent server (including acloud server) connected via the Internet or a similar network, or may beinstalled and driven in the UI device 120.

FIG. 4 shows how voice/biological information feature vectors are inputto the emotion-biological information model when these vectors arelearned. Schematically, ‘extraction of feature vectors’ starts when thecircumstances (behavior, time, etc.) are matched in the registeredconcerned context, and ‘addition of generated feature vectors to theemotion-biological information model’ is implemented when the occurrenceof the concerned context is recognized. Detailed descriptions thereofare as follows.

When it is recognized that attributes of circumstantial elementsdesignated in concerned context c_(k) registered by a user matchattributes of circumstantial elements extracted by the UI device 120,user experience (UX) data is extracted as the circumstance andvoice/biological information feature vectors in units of sliding windowWin_(thr) and transmitted to the emotion-biological information modelmanagement module 130. Thus, the emotion-biological information modelmanagement module 130 determines the current biological informationrange based on the voice/biological information feature vectorsreconfigured by considering a correlation between a previously learnedemotion-biological information model and a circumstance, a behavior andan atmospheric temperature, thereby recognizing a user's emotionalstate. The occurrence probability of the registered concerned context iscalculated to generate a corresponding concerned context event of whenthe occurrence of the concerned context is predicted or recognized, andcollected voice/biological information feature vectors V_(n+2) andV_(n+3) are managed as added to a model for learning theemotion-biological information model when the occurrence of theconcerned context is recognized. That is, referring to FIG. 4, thevoice/biological information feature vectors V_(n+2) and V_(n+3) areadded to the emotion-biological information model (see the hatchedportions), but a feature vector is not added to the emotion-biologicalinformation model because the occurrence of the concerned context is notrecognized yet. However, the description with reference to FIG. 4 ismerely one example, and may alternatively be carried out as intended bythose skilled in the art.

Referring back to FIG. 3, the concerned-context event generator 234 ofthe emotion-biological information model management module 130 generatesthe occurrence prediction or recognition event as shown in FIG. 4, andthe generated event is transmitted to the UI device 120 and/or theexternal device/service 140 (see FIG. 1) registered by a user withregard to the corresponding concerned context. The event is configuredto distinguish an identifier of the concerned context, an occurrenceprediction or recognition time, and types of prediction event orrecognition event.

FIG. 5 shows a UI screen for registering or modifying concerned contextin the UI device 120 (e.g., in the concerned context definer 221). A UIfor managing the concerned context may be synchronized with theemotion-biological information model management module 130.

Through the interface shown in FIG. 5, a user may set a concernedcontext label (310) to be registered or modified (e.g., a concernedcontext about the user's own emotion, biological information, etc., forinstance, ‘angry at conference’, ‘blood pressure goes up in themorning’). Further, among the circumstances of the correspondingconcerned context, a time slot and place (320), a user behavior (330),ranges of biological information (or a biological signal) and attributesof an emotion (340) can be set. Further, through the emotion-biologicalinformation model, a user may register first feedback to be given whenthe occurrence of the registered concerned context is predicted (i.e.,in response to a prediction event) and second feedback to be given whenthe occurrence of the registered concerned context is recognized (i.e.,in response to a recognition event) (350), and register an externaldevice/service to be used for an interface to receive the feedback whenthe occurrence prediction or recognition event is generated (360).

FIGS. 6A and 6B are respectively functional flowcharts between the UIdevice 120 and the emotion-biological information model managementmodule 130 in the apparatus and method for predicting/recognizingoccurrence of personal concerned context according to one embodiment ofthe present invention. Further, as described above, FIGS. 6A and 6B shownot only a process flow diagram in terms of a method, but also functionimplementing blocks of hardware or software in terms of an apparatus.

A user may register/modify or delete the concerned context through aconcerned context UI (e.g., refer to the interface of FIG. 5) of the UIdevice 120 (421). When a user registers/modifies or deletes theconcerned context information, the emotion-biological information modelmanagement module 130 updates corresponding concerned contextinformation in the emotion-biological information model (431).

The UI device 120 extracts current circumstances such as a time, aplace, a user behavior, an atmospheric temperature, etc. (422), anddetermines whether concerned context matching the designatedcircumstance has been registered (423). As described above, thecircumstances may be extracted through a sensor and various applicationsin the UI device 120, and may be acquired from a user wearable device(for example, the circumstance collector 212 of FIG. 3).

When it is recognized (or determined) that the circumstance designatedin the registered concerned context matches the current extractedcircumstance, a user voice feature and/or a biological informationfeature is extracted (424). As described above, the user voice featureis extractable from a voice acquired through the microphone of the UIdevice 120, and the biological information feature is extractable from abiological signal acquired from the user wearable device (for example,the biological information collector 211 of FIG. 3). Along with thecircumstance, the extracted voice feature and/or biological informationfeature are transmitted to the emotion-biological information modelmanagement module 130 (425).

The emotion-biological information model management module 130 uses thecircumstance and the voice/biological information feature received fromthe UI device 120 (425) and the reference statistics of the informationwhere user voice/biological information features related to theatmospheric temperature and the behavior have been previouslyaccumulated, thereby determining each attribute range of the userbiological information (432), recognizing a current user emotionalstate, and calculating the concerned-context occurrence probability(433). In this stage (or element) 433 for recognizing a user's emotionand calculating the concerned-context occurrence probability, thevoice/biological information feature vector is configured to includeappended information using the reference statistics of each biologicalinformation managed according to the atmospheric temperature and thebehavior in the emotion-biological information model management module130. For example, the voice/biological information feature received fromthe UI device 120 may be compared with the reference statistics of thevoice/biological information and reconfigured as the feature vectorincluding appended information such as difference, maximum (max),minimum (min), etc. Using the feature information of the already learnedemotion-biological information model and the reconfigured featurevector, a user's emotion is recognized in a corresponding concernedcontext, and the attributes of the emotion elements in the concernedcontext are determined. The probability information to be classifiedinto a specific emotion calculated in this emotion recognizing processmay be reflected in an occurrence probability about the concernedcontext. In this case, a specific method of recognizing the emotion andcalculating the probability through the feature vector and thepreviously learned model may include machine learning, knowledge basegraph model, or combination thereof.

The occurrence probability P(c_(k)) of the concerned context c_(k) iscalculated by the emotion-biological information model management module130, using the recognized emotional state information and the biologicalinformation range based on the previously learned emotion-biologicalinformation model, and using the circumstance and user voice/biologicalinformation feature vector may be obtained by the following Equation 1.

$\begin{matrix}{{P\left( c_{k} \right)} = {{{P\left( e_{j} \right)} \star {{tf}_{e}\left( {01} \right)} \star \omega_{e}} + {\sum_{n = 1}^{N}\left( {{{fs}\left( s_{n} \right)} \star {{tf}_{s_{n}}\left( {01} \right)} \star \omega_{s_{n}}} \right)} + {\sum_{m = 1}^{M}\left( {{{fb}\left( b_{m} \right)} \star {{tf}_{b_{m}}\left( {01} \right)} \star \omega_{b_{m}}} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

In brief, the Equation 1 means that a specific concerned-contextoccurrence probability P(c_(k)) is calculated by considering all of 1)probability P(θ_(j)) that the concerned context designated with aspecific emotion element will occur, 2) a function value fs(S_(n))reflecting whether attributes of circumstances in the concerned contextdesignated by a user match attributes of extracted circumstances, and 3)a function value fb(b_(m)) reflecting whether attributes of biologicalinformation in the concerned context designated by the user matchattributes of extracted biological information. In the first termtf_(e)(0|1) indicates whether a user designates the attributes of theemotion in the concerned context (e.g., 0 indicates no designation, and1 indicates a designation), and ω_(e) indicates a weight of importanceof the emotional attributes. In the second term, tf_(sn)(0|1) indicateswhether a user designates the circumstance in the concerned context(e.g., 0 indicates no designation, and 1 indicates a designation), andω_(sn), indicates a weight of importance of the circumstance. Sincethere are a plurality of circumstantial attributes, variables about thecircumstances are all added up (Σ). In the third term, tf_(bm)(0|1)indicates whether a user designates the biological response informationin the concerned context (e.g., 0 indicates no designation, and 1indicates a designation), and ω_(bm) indicates a weight of importance ofthe biological information. Since there are also a plurality ofattributes of biological response information, variables about thebiological response information are all added up (Σ).

Variables used in the Equation 1 for calculating the occurrenceprobability P(c_(k)) of the concerned context c_(k) are tabulated in aTable 3.

TABLE 3 P(c_(k)) occurrence probability of specific concerned context(c_(k)) P(θj) probability that an emotional label (θj) will berecognized in the specific concerned context (c_(k)) tf_(e)(0|1) whetheran emotional attribute is designated in the specific concerned context(c_(k)) ω_(e) an emotional weight of emotional element in the specificconcerned context (c_(k)) fs(S_(n)) a function value reflecting whetheran attribute of the circumstantial element (S_(n)) in the specificconcerned context (c_(k)) matches an attribute of the extractedcircumstantial element (S_(n)) tf_(sn)(0|1) whether an attribute of thecircumstantial element (S_(n)) is designated in the specific concernedcontext (c_(k)) ω_(sn) a weight of the circumstantial element (S_(n)) inthe specific concerned context (c_(k)) fb(b_(m)) a function valuereflecting whether an attribute of biological information element(b_(m)) in the specific concerned context (c_(k)) matches an attributeof recognized biological information element (b_(m)) tf_(bm)(0|1)whether the attribute of the biological information element (b_(m)) isdesignated in the concerned context (c_(k)) ω_(bm) a weight of thebiological information element (b_(m)) in the specific concerned context(c_(k))

Referring back to FIGS. 6A and 6B, the emotion-biological informationmodel management module 130 also updates the reference statistics aboutthe voice/biological information features corresponding to theatmospheric temperature and behavior managed therein (i.e., in theemotion-biological information model management module 130) using thevoice/biological information features, which are received from the UIdevice 120 when the circumstances in the concerned context are matched(434).

Next, it is determined whether the concerned-context occurrenceprobability calculated in the stage (or element) 433 for calculating theuser emotion recognition and the concerned-context occurrenceprobability is higher than or equal to the threshold thr₁ to such anextent as to predict the occurrence of the concerned context, or ishigher than or equal to the threshold thr₂ to such an extent as torecognize the occurrence of the concerned context (435). If determinedas such, the concerned context event (i.e., the concerned contextoccurrence prediction event or the concerned context occurrencerecognition event) is generated, and transmitted to the UI device 120and/or interfaced with the registered external device/service 140 (seeFIG. 1) (436).

Meanwhile, when the concerned context occurrence recognition event isgenerated, the previously extracted voice/biological information featurevector is added to the emotion-biological information model as thevoice/biological information feature vector corresponding to therecognized emotion label and the biological information rangecorresponding to the behavior of the circumstance of the correspondingconcerned context and used in relearning the model (437), and therelearning results are transmitted to the UI device 120 so that varioustypes of feedback registered by a user can be implemented in the UIdevice 120 or the device worn on the user (e.g., the informationcollection device 110 of FIG. 2)(426).

FIG. 7 is a semantic configuration view of an emotion-biologicalinformation model according to the present invention. Theemotion-biological information model shown in FIG. 7 refers to not atangible physical configuration but a semantic configuration fordescribing the meaning or function of the learning model achievable by amachine learning technique. Through the semantic configuration of FIG.7, an emotion-biological information learning model of variousarchitectures, such as a convolutional neural network (CNN), a recurrentneural network (RNN), deep learning, a support vector machine, etc. maybe materialized by those skilled in the art.

This emotion-biological information model is configured to have acorrelation with circumstances 520, such as a time slot, a place, anatmospheric temperature, and a user behavior according to a plurality ofregistered pieces of concerned context 510, a biological informationrange and recognized emotion label 540, and a set of voice/biologicalinformation feature vectors 550. This model manages statistics 530 of areference voice/biological information feature according tobehavior/atmospheric temperatures to reflect information about thecorrelation with the user behavior and the atmospheric temperature so asto correctly recognize the emotion based on the circumstance and thevoice/biological information feature. The circumstances (the behavior,etc.) and the voice/biological information features 560 received fromthe terminal (for example, the UI device 120) are configured again intothe feature vectors set 550 by using the biological informationreference statistics 530, and a user emotional state is recognized basedon this previously learned model.

The recognition of the user emotional state is determined in theemotion-biological information model from the voice and the biologicalresponse information, and an actual emotional state is recognizedthrough the emotion-biological information model when a user designatesthe emotional attributes of the emotional elements such as joy and angerin a specific concerned context. Moreover, in the emotion-biologicalinformation model, a user's usual emotional change features according tohis/her behavior and atmospheric temperature are stored as referencedata, and the emotional state is predicted by referring to this userdata when data, a behavior, an atmospheric temperature and biologicalresponse information are actually input. Accordingly, robust emotionalstate recognition is possible with regard to a circumstance such as auser behavior, etc.

As described above, points in time when UX data is actually collectedand processed are automatically determined according to the attributesof elements in the registered concerned context. That is, a user'svoice/biological information features are extracted without userintervention when the circumstances of the registered concerned contextare matched, and the feature vector including the voice/biologicalinformation feature, extracted when the occurrence of the concernedcontext is recognized, is added to the emotion-biological informationmodel.

Further, feedback, which is desired to be given when the occurrence ofthe corresponding concerned context is predicted or recognized, and anexternal device/service, which is desired to interface with acorresponding event, can be registered by a user as s/he wants.

In conclusion, robust emotion recognition is performed with regard toeffects of a user's specific behavior and atmospheric temperature, andan event is generated as the occurrence of the registered concernedcontext is predicted or recognized, and thus the user can receive properUI/UX from various devices and services.

Detailed embodiments of the present invention have been described aboveby way of example. However, the technical scope of the present inventionis not limited by these embodiments, but defined in rationalinterpretation of the appended claims.

What is claimed is:
 1. An apparatus for predicting/recognizingoccurrence of personal concerned context, the apparatus comprising: auser interface device comprising a concerned context definer throughwhich a user designates biological information, emotions, andcircumstances and registers concerned context about the user's ownbiological information change or emotional state; and anemotion-biological information model management module comprising anemotion-biological information model manager which reflects informationabout the concerned context registered by the user in anemotion-biological information model, makes the emotion-biologicalinformation model learn from a user voice/biological information featureand reference statistics of the user voice/biological informationfeature depending on user behavior and atmospheric temperature, andmanages the emotion-biological information model, and aconcerned-context event generator which generates a concerned-contextoccurrence prediction event or a concerned-context occurrencerecognition event by predicting or recognizing the occurrence of theconcerned context registered in the concerned context definer of theuser interface device.
 2. The apparatus of claim 1, wherein the userinterface device further comprises: a circumstance extracting andrecognizing unit which extracts a current circumstance from the userinterface device, and recognizes whether concerned context designatedwith a circumstance matching the extracted current circumstance has beenregistered; and a voice/biological information feature extractor whichextracts a voice/biological information feature from current uservoice/biological information and transmits the extractedvoice/biological information feature along with the circumstance to theemotion-biological information model management module when thecircumstance extracting and recognizing unit recognizes that the currentcircumstance matches the circumstance of the registered concernedcontext.
 3. The apparatus of claim 1, wherein the user interface devicefurther comprises: a circumstance extracting and recognizing unit whichextracts a current circumstance from a circumstance received from a userwearable information collection device, and recognizes whether concernedcontext designated with a circumstance matching the extracted currentcircumstance has been registered; and a voice/biological informationfeature extractor which extracts a voice/biological information featurefrom current user voice/biological information and transmits theextracted voice/biological information feature along with thecircumstance to the emotion-biological information model managementmodule when the circumstance extracting and recognizing unit recognizesthat the current circumstance matches the circumstance of the registeredconcerned context.
 4. The apparatus of claim 1, wherein the userinterface device further comprises: a feedback registration unit whichregisters a type of feedback desired to be given when theconcerned-context occurrence prediction or recognition event isgenerated; and a feedback giver which gives the registered type offeedback when the concerned-context occurrence prediction or recognitionevent is received from the emotion-biological information modelmanagement module.
 5. The apparatus of claim 4, wherein the userinterface device further comprises an external device/serviceregistration unit which registers an external device and service tointerface with the feedback given when the concerned-context occurrenceprediction or recognition event is generated.
 6. The apparatus of claim2, wherein the emotion-biological information model management modulefurther comprises: an emotion recognizer which extracts a feature vectorto be used for recognizing the user's emotion from the voice/biologicalinformation feature received from the user interface device; and aconcerned-context occurrence probability calculator which calculates anoccurrence probability of the concerned context on the basis of theemotion-biological information model and the extracted voice/biologicalinformation feature vector.
 7. The apparatus of claim 1, wherein theconcerned-context event generator of the emotion-biological informationmodel management module uses a threshold for determining occurrenceprediction and a threshold for determining occurrence recognition todetermine either of concerned-context occurrence prediction orconcerned-context occurrence recognition.
 8. An apparatus forpredicting/recognizing occurrence of personal concerned context, theapparatus comprising: a concerned context definer through which a userdesignates biological information, emotions, and circumstances andregisters concerned context about the user's own biological informationchange or emotional state; an emotion-biological information modelmanager which reflects information about the concerned contextregistered by the user in an emotion-biological information model, makesthe emotion-biological information model learn from a uservoice/biological information feature and reference statistics of theuser voice/biological information depending on user behavior andatmospheric temperature, and manages the emotion-biological informationmodel; and a concerned-context event generator which generates aconcerned-context occurrence prediction event or a concerned-contextoccurrence recognition event by predicting or recognizing the occurrenceof the concerned context registered in the concerned context definer. 9.The apparatus of claim 8, further comprising: a circumstance extractingand recognizing unit which extracts a current circumstance of the user,and recognizes whether concerned context designated with a circumstancematching the extracted current circumstance has been registered; and avoice/biological information feature extractor which extracts avoice/biological information feature from current user voice/biologicalinformation when the circumstance extracting and recognizing unitrecognizes that the current circumstance matches the circumstance of theregistered concerned context.
 10. The apparatus of claim 8, furthercomprising: a circumstance extracting and recognizing unit whichextracts a current circumstance from a circumstance received from a userwearable information collection device, and recognizes whether concernedcontext designated with a circumstance matching the extracted currentcircumstance has been registered; and a voice/biological informationfeature extractor which extracts a voice/biological information featurefrom current user voice/biological information when the circumstanceextracting and recognizing unit recognizes that the current circumstancematches the circumstance of the registered concerned context.
 11. Theapparatus of claim 8, further comprising: a feedback registration unitwhich registers a type of feedback desired to be given when theconcerned-context occurrence prediction or recognition event isgenerated; and a feedback giver which gives the registered type offeedback when the concerned-context occurrence prediction or recognitionevent is generated in the concerned-context event generator.
 12. Theapparatus of claim 11, further comprising an external device/serviceregistration unit which registers an external device and service tointerface with the feedback given when the concerned-context occurrenceprediction or recognition event is generated.
 13. The apparatus of claim9, further comprising: an emotion recognizer which extracts a featurevector to be used for recognizing the user's emotion from thevoice/biological information feature; and a concerned-context occurrenceprobability calculator which calculates an occurrence probability of theconcerned context on the basis of the emotion-biological informationmodel and the extracted voice/biological information feature vector. 14.The apparatus of claim 8, wherein the concerned-context event generatoruses a threshold for determining occurrence prediction and a thresholdfor determining occurrence recognition to determine either ofconcerned-context occurrence prediction or concerned-context occurrencerecognition.
 15. A method of predicting/recognizing occurrence ofpersonal concerned context, the method comprising: designating, by auser, biological information, emotions, and circumstances andregistering concerned context about the user's own biologicalinformation change or emotional state; reflecting information about theconcerned context registered by the user in an emotion-biologicalinformation model, making the emotion-biological information model learnfrom at least one of user voice and biological information features andreference statistics of user voice/biological information feature, andmanaging the emotion-biological information model; and generating aconcerned-context occurrence prediction event or a concerned-contextoccurrence recognition event by predicting or recognizing the occurrenceof the registered concerned context.
 16. The method of claim 15, furthercomprising: extracting a current circumstance of the user, andrecognizing whether concerned context designated with a circumstancematching the extracted current circumstance has been registered; andextracting a voice/biological information feature from current uservoice/biological information when it is recognized by the extraction andrecognition of the circumstance that the current circumstance matchesthe circumstance of the registered concerned context.
 17. The method ofclaim 16, further comprising: extracting a feature vector to be used forrecognizing the user's emotion from the voice/biological informationfeature; and calculating an occurrence probability of the concernedcontext on the basis of the emotion-biological information model and theextracted voice/biological information feature vector.
 18. The method ofclaim 15, wherein the generation of the concerned-context occurrenceprediction or recognition event further comprises giving feedback to theuser when the concerned-context occurrence prediction or recognitionevent is generated.
 19. The method of claim 15, wherein the generationof the concerned-context occurrence prediction or recognition eventfurther comprises using a threshold for determining occurrenceprediction and a threshold for determining occurrence recognition todetermine either of concerned-context occurrence prediction orconcerned-context occurrence recognition.